Skip to main content
SpringerLink
Log in

Hierarchical multi-resolution mesh networks for brain decoding

  • Original Research
  • Published:
Brain Imaging and Behavior Aims and scope Submit manuscript

Abstract

Human brain is supposed to process information in multiple frequency bands. Therefore, we can extract diverse information from functional Magnetic Resonance Imaging (fMRI) data by processing it at multiple resolutions. We propose a framework, called Hierarchical Multi-resolution Mesh Networks (HMMNs), which establishes a set of brain networks at multiple resolutions of fMRI signal to represent the underlying cognitive process. Our framework, first, decomposes the fMRI signal into various frequency subbands using wavelet transform. Then, a brain network is formed at each subband by ensembling a set of local meshes. Arc weights of each local mesh are estimated by ridge regression. Finally, adjacency matrices of mesh networks obtained at different subbands are used to train classifiers in an ensemble learning architecture, called fuzzy stacked generalization (FSG). Our decoding performances on Human Connectome Project task-fMRI dataset reflect that HMMNs can successfully discriminate tasks with 99% accuracy, across 808 subjects. Diversity of information embedded in mesh networks of multiple subbands enables the ensemble of classifiers to collaborate with each other for brain decoding. The suggested HMMNs decode the cognitive tasks better than a single classifier applied to any subband. Also mesh networks have a better representation power compared to pairwise correlations or average voxel time series. Moreover, fusion of diverse information using FSG outperforms fusion with majority voting. We conclude that, fMRI data, recorded during a cognitive task, provide diverse information in multi-resolution mesh networks. Our framework fuses this complementary information and boosts the brain decoding performances obtained at individual subbands.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. mathworks.com/matlabcentral/fileexchange/9554-a-numerical-tour-of-signal-processing

References

  • Alkan, S., & Yarman-Vural, F.T. (2015). Ensembling brain regions for brain decoding. In 2015 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC) (pp. 2948–2951).

  • Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., Nolan, D., Bryant, E., Hartley, T., Footer, O., Bjork, J.M., Poldrack, R., Smith, S., Johansen-Berg, H., Snyder, A.Z., & Essen, D.C.V. (2013). Function in the human connectome: task-fmri and individual differences in behavior. NeuroImage, 80, 169–189.

    Article  PubMed  PubMed Central  Google Scholar 

  • Behroozi, M., & Daliri, M.R. (2014). Predicting brain states associated with object categories from fmri data. Journal of Integrative Neuroscience, 13(04), 645–667.

    Article  PubMed  Google Scholar 

  • Behroozi, M., & Daliri, M.R. (2015). Rdlpfc area of the brain encodes sentence polarity: a study using fmri. Brain Imaging and Behavior, 9(2), 178–189.

    Article  PubMed  Google Scholar 

  • Behroozi, M., Daliri, M.R., & Boyaci, H. (2011). Statistical analysis methods for the fmri data. Basic and Clinical Neuroscience, 2(4), 67–74.

    Google Scholar 

  • Binder, J.R., Gross, W.L., Allendorfer, J.B., Bonilha, L., Chapin, J., Edwards, J.C., Grabowski, T.J., Langfitt, J.T., Loring, D.W., Lowe, M.J., Koenig, K., Morgan, P.S., Ojemann, J.G., Rorden, C., Szaflarski, J.P., Tivarus, M.E., & Weaver, K.E. (2011). Mapping anterior temporal lobe language areas with fmri: a multicenter normative study. NeuroImage, 54(2), 1465–1475.

    Article  PubMed  Google Scholar 

  • Buckner, R.L., Krienen, F.M., Castellanos, A., Diaz, J.C., & Yeo, B.T.T. (2011). The organization of the human cerebellum estimated by intrinsic functional connectivity. Journal of Neurophysiology, 106(5), 2322–2345.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bullmore, E., Fadili, J., Maxim, V., Şendur, L., Whitcher, B., Suckling, J., Brammer, M., & Breakspear, M. (2004). Wavelets and functional magnetic resonance imaging of the human brain. NeuroImage, 23(Supplement 1), S234–S249.

    Article  PubMed  Google Scholar 

  • Cabral, C., Silveira, M., & Figueiredo, P. (2012). Decoding visual brain states from fmri using an ensemble of classifiers. Pattern Recognition, 45(6), 2064–2074.

    Article  Google Scholar 

  • Castelli, F., Happé, F., Frith, U., & Frith, C. (2000). Movement and mind: a functional imaging study of perception and interpretation of complex intentional movement patterns. NeuroImage, 12(3), 314–325.

    Article  PubMed  CAS  Google Scholar 

  • Chen, M., Han, J., Hu, X., Jiang, X., Guo, L., & Liu, T. (2014). Survey of encoding and decoding of visual stimulus via fmri: an image analysis perspective. Brain Imaging and Behavior, 8(1), 7–23.

    Article  PubMed  PubMed Central  Google Scholar 

  • Cox, D.D., & Savoy, R.L. (2003). Functional magnetic resonance imaging (fmri) “brain reading”: detecting and classifying distributed patterns of fmri activity in human visual cortex. NeuroImage, 19(2), 261–270.

    Article  PubMed  Google Scholar 

  • Daliri, M.R. (2012). Predicting the cognitive states of the subjects in functional magnetic resonance imaging signals using the combination of feature selection strategies. Brain Topography, 25(2), 129–135.

    Article  PubMed  Google Scholar 

  • Delgado, M.R., Nystrom, L.E., Fissell, C., Noll, D.C., & Fiez, J.A. (2000). Tracking the hemodynamic responses to reward and punishment in the striatum. Journal of Neurophysiology, 84(6), 3072–3077.

    Article  PubMed  CAS  Google Scholar 

  • Dinov, I.D., Boscardin, J.W., Mega, M.S., Sowell, E.L., & Toga, A.W. (2005). A wavelet-based statistical analysis of fmri data. Neuroinformatics, 3(4), 319–342.

    Article  PubMed  Google Scholar 

  • Ekman, M., Derrfuss, J., Tittgemeyer, M., & Fiebach, C.J. (2012). Predicting errors from reconfiguration patterns in human brain networks. Proceedings of the National Academy of Sciences, 109(41), 16,714–16,719.

    Article  Google Scholar 

  • Fan, Y. (2003). On the approximate decorrelation property of the discrete wavelet transform for fractionally differenced processes. IEEE Transactions on Information Theory, 49(2), 516–521.

    Article  Google Scholar 

  • Fang, J., Hu, X., Han, J., Jiang, X., Zhu, D., Guo, L., & Liu, T. (2015). Data-driven analysis of functional brain interactions during free listening to music and speech. Brain Imaging and Behavior, 9(2), 162–177.

    Article  PubMed  Google Scholar 

  • Fornito, A., Harrison, B.J., Zalesky, A., & Simons, J.S. (2012). Competitive and cooperative dynamics of large-scale brain functional networks supporting recollection. Proceedings of the National Academy of Sciences, 109 (31), 12,788–12,793.

    Article  Google Scholar 

  • Glasser, M.F., Sotiropoulos, S.N., Wilson, J.A., Coalson, T.S., Fischl, B., Andersson, J.L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J.R., et al. (2013). The minimal preprocessing pipelines for the human connectome project. NeuroImage, 80, 105–124.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hariri, A.R., Tessitore, A., Mattay, V.S., Fera, F., & Weinberger, D.R. (2002). The amygdala response to emotional stimuli: a comparison of faces and scenes. NeuroImage, 17(1), 317–323.

    Article  PubMed  Google Scholar 

  • Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5), 679–685.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Kauppi, J.P., Jääskeläinen, I.P., Sams, M., & Tohka, J. (2010). Inter-subject correlation of brain hemodynamic responses during watching a movie: localization in space and frequency. Frontiers in Neuroinformatics, 4(5).

  • Kauppi, J.P., Pajula, J., & Tohka, J. (2014). A versatile software package for inter-subject correlation based analyses of fmri. Frontiers in Neuroinformatics, 8(2).

  • Kuncheva, L.I. (2004). Combining pattern classifiers: methods and algorithms. Wiley.

  • Kuncheva, L.I., Rodríguez, J J, Plumpton, C.O., Linden, D.E.J., & Johnston, S.J. (2010). Random subspace ensembles for fmri classification. IEEE Transactions on Medical Imaging, 29(2), 531–542.

    Article  PubMed  Google Scholar 

  • Lindquist, M.A. (2008). The statistical analysis of fmri data. Statistical Science, 23(4), 439–464.

    Article  Google Scholar 

  • Mandelbrot, B.B. (1977). The fractal geometry of nature. New York: Springer.

    Google Scholar 

  • Onal, I., Ozay, M., & Yarman Vural, F. (2015a). Modeling voxel connectivity for brain decoding. In PRNI, Stanford, CA, USA (pp. 5–8).

  • Onal, I., Ozay, M., & Yarman Vural, F.T. (2015b). Functional mesh model with temporal measurements for brain decoding. In Engineering in medicine and biology society (EMBC), 2015 37th annual international conference of the IEEE (pp. 2624–2628): IEEE.

  • Ozay, M., & Yarman-Vural, F.T. (2016). Hierarchical distance learning by stacking nearest neighbor classifiers. Information Fusion, 29, 14–31.

    Article  Google Scholar 

  • Richiardi, J., Eryilmaz, H., Schwartz, S., Vuilleumier, P., & Ville, D.V.D. (2011). Decoding brain states from fmri connectivity graphs. NeuroImage, 56(2), 616–626. Multivariate Decoding and Brain Reading.

    Article  PubMed  Google Scholar 

  • Richiardi, J., Achard, S., Bunke, H., & De Ville, D.V. (2013). Machine learning with brain graphs: predictive modeling approaches for functional imaging in systems neuroscience. IEEE Signal Processing Magazine.

  • Shirer, W.R., Ryali, S., Rykhlevskaia, E., Menon, V., & Greicius, M.D. (2011). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex, 22(1), 158–165.

    Article  PubMed  Google Scholar 

  • Smith, R., Keramatian, K., & Christoff, K. (2007). Localizing the rostrolateral prefrontal cortex at the individual level. NeuroImage, 36(4), 1387–1396.

    Article  PubMed  Google Scholar 

  • Thompson, W.H., & Fransson, P. (2015). The frequency dimension of fmri dynamic connectivity: network connectivity, functional hubs and integration in the resting brain. NeuroImage, 121, 227–242.

    Article  PubMed  Google Scholar 

  • Van De Ville, D., Blu, T., & Unser, M. (2006). Surfing the brain—an overview of wavelet-based techniques for fMRI data analysis. IEEE Engineering in Medicine and Biology Magazine, 25(2), 65–78.

    Article  Google Scholar 

  • Wheatley, T., Milleville, S.C., & Martin, A. (2007). Understanding animate agents: distinct roles for the social network and mirror system. Psychological Science, 18(6), 469–474.

    Article  PubMed  Google Scholar 

  • Xu, Z., & Chan, A.K. (2002). Encoding with frames in mri and analysis of the signal-to-noise ratio. IEEE Transactions on Medical Imaging, 21(4), 332–342.

    Article  PubMed  Google Scholar 

Download references

Funding

Itir Onal Ertugrul is funded by TUBITAK 2211E. This work was supported by CREST, JST, and TUBITAK Project No 116E091.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Middle East Technical University, Ankara, Turkey

    Itir Onal Ertugrul & Fatos T. Yarman Vural

  2. Graduate School of Information Sciences, Tohoku University, Sendai, Miyagi, Japan

    Mete Ozay

Authors
  1. Itir Onal Ertugrul
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mete Ozay
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fatos T. Yarman Vural
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Itir Onal Ertugrul.

Ethics declarations

Conflict of interests

Itir Onal Ertugrul, Mete Ozay and Fatos Yarman Vural declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors. Data used in this study were previously collected and archived in a data bank.

Additional information

This work was supported by CREST, JST, Grant Number JPMJCR14D1, the ImPACT Program of the Council for Science, Technology, and Innovation (Cabinet Office, Government of Japan) and TUBITAK Project No 116E091. Itir Onal Ertugrul was supported by TUBITAK 2211E.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Onal Ertugrul, I., Ozay, M. & Yarman Vural, F.T. Hierarchical multi-resolution mesh networks for brain decoding. Brain Imaging and Behavior 12, 1067–1083 (2018). https://doi.org/10.1007/s11682-017-9774-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11682-017-9774-z

Keywords

Search

Navigation